Optimization Techniques for “Scaling Down ” Hadoop on Multi-Core, Shared-Memory Systems

The underlying assumption behind Hadoop and, more generally, the need for distributed processing is that the data to be analyzed cannot be held in memory on a single machine. Today, this assump-tion needs to be re-evaluated. Although petabyte-scale datastores are increasingly common, it is unclear whether “typical ” analyt-ics tasks require more than a single high-end server. Additionally, we are seeing increased sophistication in analytics, e.g., machine learning, where we process smaller and more refined datasets. To address these trends, we propose “scaling down ” Hadoop to run on multi-core, shared-memory machines. This paper presents a proto-type runtime called Hone (“Hadoop One”) that is API compatible with Hadoop. With Hone, we can take an existing Hadoop ap-plication and run it efficiently on a single server. This allows us to take existing MapReduce algorithms and find the most suitable runtime environment for execution on datasets of varying sizes. For dataset sizes that fit into memory on a single machine, our experi-ments show that Hone is substantially faster than Hadoop running in pseudo-distributed mode. In some cases, Hone running on a sin-gle machine outperforms a 16-node Hadoop cluster. 1